Projection Scanning System

ABSTRACT

Imaging systems projecting augmented information on a physical object that at a minimum include a processor, a memory device operably connected to the processor, a projector operably coupled to the processor, and a distance-measuring device operably connected to the processor. The memory device stores augmented image information, and the processor is configured to project augmented image information onto the physical object. The distance-measuring device is configured to measure the distance to the physical object. The processor uses distance measurement information from the distance measuring device to adjust scaling of the augmented image information. The processor provides the scale adjusted augmented image information to the projector. System can also be used for fluorescence imaging during open surgery, for endoscopic fluorescence imaging and for registration of surgical instruments.

CROSS-REFERENCE

This application is a continuation of U.S. application Ser. No.16/634,051, filed Jan. 24, 2020, which is a U.S. National PhaseApplication under 35 U.S.C. § 371 of International Application No.PCT/US2018/044169, filed Jul. 27, 2018, which claims the benefit of U.S.Provisional Application No. 62/537,627, filed Jul. 27, 2017, whichapplications are entirely incorporated herein by reference.

BACKGROUND

The present application generally relates to medical systems, devices,and methods, and more particularly relates to surgical imaging andprojection.

Oftentimes, it can be challenging to present visual information to asurgeon during a surgical procedure. For example, visual informationpresented on an external monitor may require the surgeon to disrupt hisor her focus on a surgical site to view the external monitor, and viceversa. This may be undesirable such as when the surgeon would prefer tointake the visual information whilst maintaining focus on the surgicalsite, or when the surgeon is otherwise unavailable to remove his or hergaze from the surgical site.

SUMMARY

Disclosed is an improved surgical imaging and projection system. Thesystem of the present invention provides augmented information byprojecting visual data onto a target. The system of the presentinvention includes a feedback loop that automatically adjusts the scaleof the projected visual information to match the scale of the target. Insome embodiments, the system detects a fiducial on the target andregisters the projected visual information with the target using thelocation of the fiducial. The system is particularly useful forprojecting additional surgical information onto a surgical site. Theprojected visual information may also be termed augmented or false colorimage, and is projected onto and registered with the real object such asa surgical site.

Many video image processing systems, such as X-ray, ultrasound, MRI orfluorescent systems are used during surgery. These image systemstypically utilize a monitor to provide the surgeon with additionalinformation. Although this information is helpful, viewing thisadditional data requires the surgeon to shift his/her field of viewbetween the surgical field and a monitor. The present inventiondiscloses embodiments for projecting the augmented image informationdirectly onto the patient (surgical field). The augmented imageinformation can be CT images (computerized axial tomography), MRI images(magnetic resonance imaging), fluorescence information or the like. Thisenables the surgeon to maintain his/her view of the surgical field andis completely intuitive.

According to one embodiment a false color image (in the visible lightspectrum), corresponding to an augmented image (e.g., fluorescent imageor MRI image), is projected directly onto the patient (surgical field).This enables the surgeon to “see” the augmented image without the use ofspecial equipment (e.g., goggles) while maintaining their field of viewon the surgical field. However, one of the embodiments disclosed hereinincludes a head mounted display, which may be advantageous in certainsettings.

One of the problems that has prevented the projection of augmented imageinformation in a surgical setting relates to scaling and registration ofthe augmented image relative to the three dimensional human anatomicalstructures. When the camera/projector is at a specific distance relativeto the surgical field it will image and project a picture at a specificmagnification. It is critical that the projected image is registeredwith the actual real anatomical object, i.e., size and orientation. Thismay be easily optimized for a fixed, known distance to adjust themagnification of the projected image. However, the issue arises when thedistance or orientation of the camera changes from that optimizedposition. If the camera moves further away from the object, the objectwill appear smaller to the camera and thus the scale of the imageprojected will be smaller than the real object.

The embodiments disclosed herein monitor the distance and orientation ofthe projector relative to the target (e.g., patient or real object) inreal time and automatically adjust the magnification/scaling of theprojected image.

Another problem that has prevented the projection of augmented imageinformation in a surgical setting relates to the orientation of thecamera relative to the body. The perceived shape of the real object willdiffer depending on the viewing angle. It is critical that theorientation of the augmented image match the orientation of the realobject.

It would be useful to provide intuitive real-time visual feedback to theuser on how to adjust the orientation of the system for optimal results.

In another part of the invention, the ability to measure distance,orientation and the actual target topology can create significantbenefits in systems that require high sensitivity to signal. Forexample, in fluorescence imaging, the goal is to create an imagingsystem with a large aperture to collect as much emission light aspossible (to maximize sensitivity). Doing so creates a challenge becausegenerally a large aperture system suffers from very short depth offocus. When the camera in such a system moves beyond the range of thedepth of focus, the lens must be refocused. This can be resolved byplacing a variable focal lens in the optical train, more specifically,it is ideal to use electrically tunable lenses. Continuous monitoring ofthe physical distance between the camera and the physical object in realtime allows the system to adjust the focal length of the variable lengthlens (in real time) while maintaining the aperture as large as possible.It is highly desirable to have the auto-focusing occur without anyvisible delay (as commonly seen with traditional mechanically drivenfocusing systems) as to minimize the image quality deterioration.

Specifically for fluorescence, the system is able to monitor theintensity signal based on distance between the camera and the target.Therefore, the system can be calibrated to quantify the amount of pixelintensity based on emission (fluorescence) signal. As will beappreciated, the intensity of the fluorescent signal varies withdistance. The shorter the distance (closer the system is to thefluorescing object), the more intense the emission signal. The systemmay be able adjust the gain of the signal in relation to the measureddistance. If the emission signal is weaker than expected for a givendistance then the system may determine that there is an issue requiringfurther attention.

Yet another problem, which has prevented the projection of augmentedimage information in a surgical setting, relates to focus across a broadarea. The human body is not, for the most part, planar. The topology ofthe human body makes it difficult to keep multiple planes in focus. Thepresent invention discloses several approaches for measuring thetopology of the physical object, which may facilitate keeping each ofthe planes in focus.

In an aspect, provided is an imaging system projecting augmentedinformation on a physical object, the imaging system comprising: aprocessor; a memory device operably connected to the processor, thememory device storing augmented image information; a projector operablycoupled to the processor, the processor configured to project augmentedimage information onto the physical object; and a distance-measuringdevice operably connected to the processor and configured to measure thedistance to the physical object, the processor using distancemeasurement information from the distance measuring device to adjustscaling of the augmented image information, the processor providing thescale adjusted augmented image information to the projector.

In some embodiments, the distance-measuring device is selected from thegroup consisting of laser range finder, laser scanning, time of flight,structured light, light field camera, and acoustic measurement device.

In some embodiments, the distance-measuring device further determines atleast one of a topology of the physical object and an orientation of thephysical object relative to the distance-measuring device, and theprocessor further uses the topology and/or orientation information toadjust scaling of the augmented information.

In some embodiments, the imaging system further comprises at least onelight source selected from the group consisting of structured light,background infrared light, visible light, and fluorophore excitationlight.

In some embodiments, the distance measuring device comprises: at leastone of an sensor and a camera operably connected to the processor,wherein the at least one light source includes a source of structuredlight configured to project a predefined light pattern onto the physicalobject in at least one of a visible and an infrared light spectrum, andwherein the at least one of a sensor and a camera detect a reflectanceof the predefined light pattern and the processor calculates at leastone of the distance to the physical object and a topology of thephysical object using the detected reflectance.

In some embodiments, the distance measuring device comprises: at leastone of a sensor and a camera operably connected to the processor,wherein the projector is configured to project a predefined lightpattern onto the physical object in at least one of a visible and aninfrared light spectrum, and wherein the at least one of a sensor and acamera detect a reflectance of the predefined light pattern and theprocessor calculates at least one of the distance to the physical objectand a topology of the physical object using the detected reflectance.

In some embodiments, the projector projects at least two predefinedlight patterns onto the physical object and the at least one of a sensorand a camera detect a reflectance of the at least two predefined lightpatterns and the processor calculates at least one of the distance tothe physical object and a topology of the physical object using thedetected reflectance.

In some embodiments, the at least one of a sensor and a camera arefurther configured to detect a location of a fiducial attached to orpainted onto the physical object, the fiducial being visible in either avisible light spectrum or an infrared spectrum, the processor using thelocation of the fiducial to register the augmented image informationwith the physical object.

In some embodiments, the distance measuring device comprises a time offlight sensor unit including a light source and a sensor. In someembodiments, the light source is an infrared capable light source andthe light sensor is capable of detecting infrared light.

In some embodiments, the distance measuring device comprises: at leastone of a sensor and a camera operably connected to the processor; and asecond projector, said second projector configured to project at leastone predefined structured light pattern in either a visible or infraredspectrum onto the physical object, wherein the at least one of a sensorand a camera detect a reflectance of the predefined light pattern, theprocessor calculates at least one of the distance to the physical objectand a topology of the physical object using the detected reflectance anduses the calculated distance and/or topology to adjust scaling of theaugmented information, the processor providing the scale adjustedaugmented information to the projector.

In some embodiments, the at least one of a sensor and a camera arefurther configured to detect a location of a fiducial attached to orpainted onto the physical object, the fiducial being visible in either avisible light spectrum or an infrared spectrum, the processor using thelocation of the fiducial to register the augmented image informationwith the physical object.

In some embodiments, the imaging system further comprises an IMUoperably connected to the processor.

In some embodiments, the processor disables the distance-measuringdevice from making distance measurements if the IMU detects that theimaging system is stationary for at least a predefined amount of time.

In some embodiments, the processor disables the projector fromprojecting if the IMU detects that the imaging system is moving morethan predefined threshold amount or misoriented.

In some embodiments, the processor triggers the projector to display analert if the imaging system is not oriented within predefined threshold.

In some embodiments, the processor triggers the projector to display analert if the measured distance to the physical object exceeds apredefined threshold distance.

In some embodiments, the imaging system further comprises a light,wherein the processor disables the light if the IMU detects that theimaging system is moving more than predefined threshold amount ormisoriented.

In some embodiments, the imaging system further comprises: at least oneof a camera or sensor operably connected to the processor and configuredto detect a fluorescent emission, the processor causing the projector toproject light to excite fluorophores in or on the physical object, andthe camera or sensor detecting a fluorescent emission of thefluorophores, the processor storing the detected fluorescent emission inthe memory as augmented image information.

In some embodiments, the projector projects the augmented imageinformation onto the physical object using light in the visiblespectrum.

In some embodiments, the projector has a first mode for projectingaugmented image information on the physical object, and second mode forprojecting light to excite fluorophores in or on the physical object. Insome embodiments, the projector has a third mode for projecting apredefined light pattern.

In some embodiments, the projector is capable of projecting light inboth a visible and an infrared light spectrum.

In some embodiments, the imaging system further comprises: at least oneof a camera or sensor operably connected to the processor and configuredto detect a fluorescent emission; and at least one of a light source anda second the projector configured to project light onto the physicalobject, the processor causing the at least one of a light source and asecond projector to project light to excite fluorophores in or on thephysical object, and the at least one of a camera or sensor detecting afluorescent emission of the fluorophores, the processor storing thedetected fluorescent emission in the memory as augmented imageinformation.

In some embodiments, the processor triggers the projector to display analert if an intensity of the detected fluorescence is below a thresholdvalue.

In some embodiments, the imaging system further comprises: at least oneof a camera or sensor operably connected to the processor; and a secondthe projector operably connected to the processor and configured toproject a virtual fiducial in either an infrared light or visible lightspectrum onto the physical object, the processor causing the secondprojector to project the virtual fiducial on the physical object, thecamera or sensor detecting a location of the virtual fiducial, and theprocessor using the location of the virtual fiducial to register theaugmented image information with the physical object.

In some embodiments, the memory is operably connected to the processorusing a remote communications protocol.

In some embodiments, the augmented image information is projected in thevisible light range.

In some embodiments, the imaging system further comprises: at least oneof a camera or sensor operably connected to the processor and configuredto detect fluorescent emission, the processor causing the projector toproject light to excite fluorophores in or on the physical object, andthe camera or sensor detecting a fluorescent emission of thefluorophores in a first wavelength, the processor storing the detectedfluorescent emission in the memory as augmented image information andtriggering the projector to project the augmented image information ontothe physical object in a second wavelength different from the firstwavelength.

Additional aspects and advantages of the present disclosure will becomereadily apparent to those skilled in this art from the followingdetailed description, wherein only illustrative embodiments of thepresent disclosure are shown and described. As will be realized, thepresent disclosure is capable of other and different embodiments, andits several details are capable of modifications in various obviousrespects, all without departing from the disclosure. Accordingly, thedrawings and description are to be regarded as illustrative in nature,and not as restrictive.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.To the extent publications and patents or patent applicationsincorporated by reference contradict the disclosure contained in thespecification, the specification is intended to supersede and/or takeprecedence over any such contradictory material.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings (also “Figure” and “Fig.” herein), of which:

FIG. 1 shows a block diagram of an augmented imaging system.

FIG. 2 shows a block diagram of a fluorescence augmented imaging system.

FIG. 3 shows a block diagram of a head mounted display imaging system.

FIG. 4 shows a block diagram of an endoscopic system.

DETAILED DESCRIPTION

While various embodiments of the invention have been shown and describedherein, it will be obvious to those skilled in the art that suchembodiments are provided by way of example only. Numerous variations,changes, and substitutions may occur to those skilled in the art withoutdeparting from the invention. It should be understood that variousalternatives to the embodiments of the invention described herein may beemployed.

As noted above, the invention is useful in providing augmented visualinformation from any source including but not limited to ultrasound,x-ray, CT scan, fluorescence imaging or the like.

Augmented Information Projection

The invention is useful in projecting augmented information onto aphysical object. The phrase “physical object” is intended to include thepatient and the surgical site, but is not so limited. In order for theaugmented information to be truly useful, it must be preciselyregistered with the physical object. In other words, the alignment, thescale and orientation must match the position and scale of the physicalobject onto which it is projected. Better yet, the augmented image canbe digitally altered to compensate for the orientation and/or topologyof the physical object.

The present invention contemplates several different ways to measure thedistance between the projector and the physical object, which areexplained at length below. Some of these distance-measuring techniquesalso yield the orientation and/or topology/contour of the physicalobject. The system of the present invention uses the calculated contourand/or distance to scale the augmented information so that itcorresponds to the scale of the physical object.

The augmented information may be saved or transmitted to the systemwirelessly or over a data cable. The wireless communication may useBluetooth, WIFI, or any other communication protocol.

Measuring the Distance Between the Camera or Projector and the PhysicalObject

A conventional laser range finder (distance sensor) may be used tomeasure the distance between the projector and the physical object. Thelaser shines light on the physical object and a sensor detects how muchtime it takes for the light to be reflected back. It is straightforwardto calculate the distance between the object and the laser emitter giventhe time it takes for the reflectance to reach the sensor and the speedof light. This technique is useful but not ideal since it does notprovide information regarding the contour/topology of the physicalobject in the field of view of the camera. One example of a commerciallaser range finder is LIDAR-Lite by Garmin International, Inc., 1200 E.151st St., Olathe, Kans. 66062-3426. There are two different mechanismsby which distance sensors measure the distance to an object, time offlight or triangulation. These two mechanisms can be achieved witheither a laser or an LED as the light source.

Techniques for Measuring the Contour/Topology

There are various ways to capture 3D topology of objects. The classicway utilizes a stereo setup where a pair of cameras is used to capturetwo different points of view. Next, image processing is used toreconstruct the image into a 3D depth map. This strategy does not workwell when there is a lack of contrast; it is also hardware and softwareintensive and is not commonly used due to its complexity.

This classic process has now evolved into four configurations: LaserScanning, Time of flight, Structured Light, and Light Field. Each ofthese techniques are applicable to the present invention. Thedifferences between these techniques are as follows:

1) Laser Scanning typically uses a single camera and a laser-projectedline. This system requires relative movement between the object and thelaser-projected line. So either the laser projected line is scanned(moved over the object) or the object is displaced to create therelative movement. As the line becomes distorted due to the topology ofthe object, the camera picks up the deformation and reconstructs theshape of the object. This is the most accurate way to scan the object,but requires either the object or the imaging system to move, which maynot be practical for static applications.

2) Time of Flight (TOF) imaging sensors are commercially available.Three-dimensional (3D) time of flight imaging sensors operate byilluminating an area with modulated IR light pattern. By measuring thephase change of the reflected signal the distance can be accuratelydetermined for every pixel in the sensor creating a 3D depth map of thesubject, scene or object. There are various companies producing time offlight cameras and sensors such as Texas Instruments, Intel, etc. Thisis an excellent method to determine the position of objects as well aslocal depth. However, this method is only accurate to within fewmillimeters and is sensitive to ambient light. So is not as accurateoutdoors, for example.

3) Structured Light uses a predefined pattern or more preferably aseries of predefined patterns, which are projected onto an object. Thepattern(s) may or may not be visible to the unaided human eye, e.g. IRpattern. The camera compares the projected pattern against the detectedpattern and determines the distance and contour/topology of the objectfrom the deformation of the detected pattern compared to the originalprojected one. The optimal method to do this is to project variouspatterns to increase the resolution of the depth map. The structuredprojection is an excellent way to scan objects with accuracy andefficiency.

4) Light Field Cameras, also known as plenoptic cameras, captureinformation about the light field emanating from a scene; that is, theintensity of light in a scene, and the direction that the light rays aretraveling in space. This contrasts with a conventional camera, whichrecords only light intensity. One type of light field camera uses anarray of micro-lenses placed in front of an otherwise conventional imagesensor to sense intensity, color, and directional information.Multi-camera arrays are another type of light field camera.

FIG. 1 represents a block diagram of the basic embodiment of anaugmented imaging system 100 according to the present invention. Theunhashed blocks, represent this basic embodiment, while hashed blocksdepict the functionalities that are optional. The augmented imagingsystem 100 includes a distance-measuring device 102, a processing unit104, and a projector 108. In this basic embodiment, processing unit 104uses the distance information from the distance-measuring device 102 toadjust the scale of the augmented image information and then direct thatinformation to the projector 108. The projector then projects the scaledaugmented image onto the object.

Device 102 is composed of an emitter and a receiver for distance ortopology measurement. It can include a variety of emitting and detectingmodalities and components such as acoustics, triangulation and rangefinder lasers, laser scanning, photodetectors and TOF. The specific waythe distance is measured is not critical as long as it is accurate. Theprocessing unit 104 includes image processing power, memory and lookuptables. The projector 108 may be a three-color (RGB) laser, an IR laseror an LED projector. Projector 108 can project continuously orintermittently in visible or IR and can also provide structured light invisible or IR. Furthermore, projector 108 can also alternate betweenaugmented image and structured light. A DLP projector may also be usedbut such a projector may not provide optimal results. A laser projectoris particularly advantageous because it can utilize the distanceinformation to optimally focus the augmented image projection inmultiple planes.

In some embodiments, the augmented information is transmitted wirelesslyto the augmented imaging system 100. Any wireless communicationsprotocol may be used. Such wireless communications protocol may includeBluetooth, WiFi, or the like. The augmented information may be anyinformation from any source. In a surgical setting, the augmentedinformation may be any information useful to the surgeon and may includea CT scan, MRI image, X-ray, ultrasound, fluorescence or otherinformation useful to the surgeon such as information for orienting asurgical tool.

In some other embodiments, distance sensor 102 may be used for measuringthe topology/contour of the object or surgical site. As an alternativeto distance sensor 102, system 100 may include the use of structuredlight to measure the distance and/or topology of the object and at leastone detector or a camera 110. Sensor/camera 110 could be a video IR,visible or hyperspectral sensor, it could also be a TOF or an acousticsensor. In this embodiment, one or more predefined patterns areprojected on the physical object by projector 108. A sensor/camera 110detects the projected pattern, and the processing unit 104 compares theprojected pattern with the detected pattern to determine thetopology/contour of the physical object. As mentioned above, thisembodiment would allow not only the topological characterization, butalso the assessment of the distance from the camera/sensor 110 orprojector 108 to the physical object.

Projector 108 Provides Structured Light

The structured light pattern may be projected in a visible lightspectrum (visible to the unaided human eye) or an IR spectrum. Moreover,the structured light may be projected using a projector 108, which insome embodiments is a visible light projector. In other embodiments, thestructure light is projected in an IR spectrum using projector 108 thatmay be capable of projecting in both a visible light spectrum and IR.That is, both visible and IR light-emitting sources may be provided onthe same module of projector 108, and both emitting sources are scannedusing the same micromirror. Alternatively, projector 108 may be used toproject the false color augmented images in the visible light spectrumand another projector may be used to project the structured light eitherin visible or in the IR spectrum.

It is also possible to use projector 108 to project both augmented imageinformation and structured light by polling or switching between anaugmented image mode and a structured light mode. In the augmented imagemode, projector 108 projects an augmented (false color) RGB image ontothe physical object. In the structured light mode, the projector 108will project one or more pre-determined structured light pattern(s) ontothe object. These structured light patterns may be projected in an IRspectrum or in a visible light spectrum.

Dedicated Structured Light

To reduce the burden on projector 108, it may be desirable to utilize adedicated source of structured light. In such an embodiment, theprojector 108 may be utilized to project the augmented image informationonto the physical object. A dedicated source of structured light 117 maybe utilized to project one or more pre-defined patterns of light ontothe physical object. Structured light 117 may include a diffractiveoptical element or a holographic film. This dedicated light source canproduce pre-defined patterns in visible or IR spectrum.

Additional Components for Topology Assessment

It should be noted that the distance measurement and topology can beobtained in a variety of ways. For example, distance measuring can beaccomplished by utilizing a distance sensor 102 that contains a laserbeam and sensor in one single compartment. Alternatively, it is alsopossible to measure the distance by using the projector 108 or lightsource 117 to project a signal or a pre-defined pattern in visible or IRand use sensor/camera 110 to assess the distance.

In any of the embodiments described herein, the distance-measuringdevice 102 may utilize a time-of-flight sensor module. The distancemeasuring device may also utilize a TOF sensor 110 that may be operablyconnected to the processing unit 104 in conjunction with a source ofstructured light such as projector 108 or light source 117. Theprojector 108 or structure light source 117 projects a structured lightpattern and the TOF sensor 110 measures the phase change of thereflected signal for every pixel in the sensor creating a 3D depth mapof the object (topology) and provides this information to the processingunit 104. The processing unit 104 uses image processing and the measuredtopology to register the image and adjust the magnified/augmented imageto properly project with correct magnification and orientation.

The processing unit 104 uses the determined topography information toadjust the projection magnification of the augmented image informationprojected by projector 108.

Distance measurement is not limited to laser light signals. Likewise,the topological characteristics of an object can be assessed usingdifferent hardware combinations, including, for example, using theprojector 108 or light source 117 to project a structured light imageand the sensor/camera 110 to capture the deformation of the patternproduced by the object.

It is also possible to use structured light to determine the distancebetween the system and the physical object. In other words, structuredlight may be used to determine the topology of the object, orientationand the distance. In such an embodiment, the components used todetermine topology also serve as the distance-measuring device 102.Measuring the distance between the system and the physical object, canbe used to adjust the focus of camera/sensor 110.

The structured light pattern may also be useful in determining therelative orientation between the system and the real object (skewangle). The camera/sensor 110 picks up the reflectance of the structuredlight pattern(s) and the processing unit 104 compares the reflectedpattern with the projected pattern and determines the orientation of thecamera/projector relative to the real object and utilizes signalprocessing to adjust the visible projection to match the orientation ofthe camera. The processing unit 104 may be used to either store orbuffer the augmented image information, and may be connected to thesystem using a data cable or may be wirelessly interfaced using WIFI,Bluetooth, or the like.

The above-described system may be attached to a stationary mount;however, the system is particularly useful as a hand held device becauseof its ability to automatically and in real-time measure and compensatefor the distance and orientation of the system relative to the objectonto which the augmented information is being projected. The hand-heldsystem of the present invention may be used as needed at various timesduring a surgical procedure. As such, there is likely to be some needfor image stabilization as the camera is moved. In such a system, anoptional stabilization module 118 may be added.

The augmented imaging system 100 of the present invention is preferablyhandheld. System 100 may optionally be enhanced by the inclusion aninertial measurement unit (IMU) 112 which may contain one or more motionand/or orientation sensor(s) such as accelerometers, gyros, andmagnetometers. It may be desirable to include the IMU 112 operablycoupled with the processing unit 104 to shut-off the projector(s) 108,or the distance/topology measuring device 102 when the detected systemorientation deviates from an established range.

Fluorescence Imaging System 200

The augmented image projection system is particularly suited for usewith a fluorescence imaging system. FIG. 2 is a block diagram offluorescence augmented imaging system 200. Like reference numbers areintended to correspond to the same structures. Unhashed blocks refer tothe basic system configuration while hashed blocks refer to optionalelements. Distance measuring device 102 may utilize structured light, aTOF sensor, an acoustic sensor, a laser scanner, or a laser range finderor the like as described above with reference to FIG. 1 . As will beexplained below in further detail, system 200 may optionally include anadditional projector 108 that may project in an infrared spectrum,visible or in both an infrared and visible light spectrum.

The system 200 may be equipped with four different and optional lightsources as follows:

1) Background Illumination (116A)—The system 200 may include lightsource 116A to provide background illumination in an IR spectrum.

2) General Illumination (116B)—The system 200 may include light source116B to provide general illumination in a visible light spectrum.

3) Fluorescent Excitation (116C)—The system 200 may include light source116C to excite fluorophores. Light source 116C may illuminate in eithera visible or IR spectrum depending on the specific needs of thefluorophores which will be explained in detail below.

4) Structured Light (117)—Similarly to System 100, system 200 mayinclude light source 117 to provide the structured light pattern(s) ineither a visible or IR spectrum. It should be understood that lightsource 117 could eliminate the need to have projector 108 project thestructured light pattern(s).

The system in FIG. 2 may optionally include an additional projector 108and any combination of light sources 116A, 116B, 116C and 117. It may bedesirable to include an IMU 112 operably coupled with the processingunit 104 to shut-off the projector(s) 108, and/or light source(s) 116A,116B, 116C, 117 when the detected system orientation deviates from anestablished range.

The IMU 112 can also be used to selectively deactivate/reactivate thedistance monitoring process. For example, if the motion sensor detectsthat the system 200 is being held in a fixed or rigid manner (e.g. on astand), the IMU 112 can disable the distance measurement process. If andwhen the IMU 112 detects that the system has moved then the processingunit 104 may resume the distance measurement process. This wouldeliminate unnecessary processing.

It may beneficial to have a fluorescent imaging system that ishand-held. Similar to system 100, system 200 may require imagestabilization in which an optional module 118 is included.

System 200 is configured to excite either intrinsic, naturally occurringfluorophores, or extrinsic, those that may be added, either by infusionor by painting them onto the surface of the object. By manner ofexample, indocyanine green is a fluorophore approved for human use andis available under the trade name IC-GREEN®. IC-GREEN® is a watersoluble, tricarbocyanine dye with a peak spectral absorption around 800nm. Some fluorophores may be configured to bind to a particular group ofcells and may be used to determine whether an area is adequatelyperfused, identify lesions, blood vessels, or the like.

Some fluorophores are excited by light within the visible spectrum andemit in either a visible or IR spectrum. Other fluorophores are excitedby infrared (IR) light and emit in either an IR or visible spectrum, thelatter through photon up-conversion.

The general principle is the same regardless of which type offluorophore is used. Namely, the fluorophores are excited using a lightsource, the emission of the fluorophores is detected, and an augmentedor false color image of detected emission is projected onto the physicalobject. In the case where the fluorophore emits in the visible range,the image information may be enhanced, as with false color, to provideadditional information.

If the fluorophores are excited by light in an IR spectrum then aprojector 108 or light source 116C may be used to provide the IR used toexcite the fluorophores.

If the fluorophores are excited by light in a visible spectrum thenprojector 108 or light source 116B may be used to provide the visiblelight used to excite the fluorophores.

If the fluorophores emit in the IR spectrum then an IR-capablecamera/sensor 110 picks up the fluorescent emission signal that isinvisible to the unaided human eye and the background illuminationsignal (if used), and stores this data in memory, processor 104.Projector 108 projects a visible light (false color) or augmentedrepresentation corresponding to the fluorescent signal onto the realobject.

If the fluorophores emit in the visible spectrum then a visible light/IRor hyperspectral capable camera/sensor 110 picks up the fluorescentemission and the background IR illumination (if used) signals. Theemission intensity and background information are processed by theprocessing unit 104, and stored as augmented/enhanced image information.Projector 108 can then project a visible light (false color) imagecorresponding to the processed/augmented fluorescent emission onto thereal object.

Thus, in some embodiments camera/sensor 110 is an IR capable sensor, inother embodiments camera/sensor 110 is a visible light capable sensor,and in yet other embodiments camera/sensor 110 is a capable of detectingboth visible and IR light, e.g., a multispectral or a hyperspectralcamera which can image both visible and IR. In yet another embodimentcamera/sensor 110 is the combination of two different sensors, one thatis more sensitive to visible wavelength signals, and one that is moresensitive to IR signals.

In embodiments using IR to excite the fluorophores, a bandpass filter(not illustrated) may be utilized to improve the signal to noise ratioby filtering out the wavelength of the IR light projected onto theobject. More particularly, the wavelength of IR used to excite thefluorophores is different from the wavelength of the fluorescentemission of the excited fluorophores. The bandpass filter may be used tofilter out the wavelength of the IR used to excite the fluorophores sothat the camera only sees the wavelength of the fluorescent emission ofthe excited fluorophores.

If the fluorophores emit in the visible light spectrum thencamera/sensor 110 picks up the emission signal. The distance sensor 102assesses the distance between the camera/sensor 110 and the fluorescingobject. The processing unit 104 can then manipulate the emission signalinformation as well as the distance and direct the projector 108.Projector 108 projects this visible light processed image (false color)onto the physical object but in a different visible spectrum that willnot interfere with the wavelength of the emission of the visible lightfluorophores. Thus, the camera 110 will use a filter (not illustrated)when capturing the emission of the visible light fluorophores to filterout the wavelength used to project the augmented image information.

The imaging system 200 may further include the capability of measuringthe intensity of the fluorescence. The low intensity of the fluorescencemay be due, for example, to the system being too far away from thefluorescence emission, i.e. the physical object being imaged. Theimaging system 200 can gauge the distance to the object with theutilization of the distance sensor 102. The intensity of thefluorescence detected by the camera/sensor 110 will increase as thesystem 200 is brought closer to the physical object. Correspondingly,the intensity of the fluorescence will decrease as the system 200 ismoved farther from the object. If the system 200 is positioned too faraway from the object the intensity of the fluorescence will degradebelow an acceptable threshold and the system will provide an alert tothe user to bring the system closer to the object. The alert can beaudible and/or visual. For example, the system 200 can project a warningdirectly on the object instructing the user to bring the system closer.

The system 200 may further include, within processing unit 104, alook-up table including the ideal threshold fluorescence intensityvalues as a function of distance. The system 200 can sense the distanceto the physical object using, for example, distance sensor 102. If themeasured fluorescence intensity is below the ideal threshold IRintensity value for the measured distance this problem may be due to lowfluorophore concentration. In that case, the user may need to applyadditional fluorophore to the physical object. The system 200 can alsoadjust the intensity of the excitation source being used to determinewhether that will enhance the detected fluorescent emission intensity.

Fiducial Targets and Head Mounted Display

In any of the embodiments described herein, it may be useful to have oneor more fiducial targets attached to one or more landmarks (e.g.,anatomical landmarks) on the physical object on which the augmentedinformation is being displayed. Such fiducial targets are useful formaking sure that the augmented image is aligned (registered) with theobject onto which it is projected. The target(s) may be physicallyattached to the object using any of a variety of known means including:an anchor inserted into the object, a belt, adhesive, sutures or thelike. The target(s) may also include preprinted stickers, or preprinted3M™ Ioban™ incise drapes that include the fiducial targets. It is alsopossible to draw the fiducial target directly on the subject/objectusing, for example, fluorescent ink.

The fiducial target may include a pattern visible to the unaided humaneye, or the pattern may be printed using infrared ink, which is onlyvisible to an IR capable camera, IR sensitive sensor or the like. The IRfiducial target has an advantage in that it will not interfere with theaugmented image projected onto the object. It is also possible toproject a virtual fiducial pattern onto the object. The virtual fiducialtarget may be projected using a visible light spectrum or morepreferably using a light spectrum not visible to the unaided human eye,e.g., infrared light so as not to interfere with the augmented image. Ifthe fiducial is detected in the IR spectrum, care should be taken toensure that the IR frequency(s) used to excite the fiducial do notinterfere with the IR emission frequencies of the fluorophores.

The fiducial target may be projected using the same projector 108 usedto project the augmented or false color image onto the object or can beproduced with structured light source 117, and it would be detected byeither camera/sensor 110 or the head mounted display 120 described belowfor system 300. In an embodiment where the fiducials are projected byprojector 108, the processing unit 104 would poll the projector 108 suchthat it would alternate between projecting the augmented imageinformation and projecting the fiducial target. Using the same projector108 to project the fiducial target and the augmented image eliminatesthe need to correct for misalignment between two distinct projectors.

There are various reasons why it may be advantageous to use two separateprojectors, with one projector 108 for projecting the fiducial targetand another projector for projecting the augmented image. In such asystem, image processing may be required to bring the two projectorsinto alignment.

As noted above, the fiducial marker or target may be projected in alight spectrum not visible to the unaided eye, e.g., infrared (IR). Thisis advantageous in that the target will not interfere with the augmentedor false color image projection.

Regardless of whether the fiducial target is printed/attached orprojected, a custom patient-specific fiducial target can be utilized.The patient-specific fiducial contains information that would be keptprivate and can only be deciphered by the camera/software of the imagingsystem.

FIG. 3 is a schematic diagram of a head mounted imaging system 300 inwhich the augmented image information is projected onto a head mounteddisplay 120. The head mounted display has integrated sensors includingimaging and inertial sensors, such as the Hololens from Microsoft.System 300 is substantially similar to system 200 (FIG. 2 ) but here theaugmented information is displayed on the head mounted display 120rather than being projected onto the object.

System 300 may include a distance-measuring device 102 as describedabove (or calculates the distance using structured light as describedpreviously), a processing unit 104 receiving distance information fromthe distance measuring device 102, and processing and managing theaugmented image, and a projector 108. In this embodiment, the projector108 may be used to project a virtual fiducial onto the object ratherthan augmented information. Alternatively, a fiducial may be attached toor painted onto the physical object. The fiducial may be created in anIR pattern invisible to the unaided human eye by structured light source117 or may be a visible target. In this embodiment, the user wears ahead mounted display 120 equipped with a camera sensor 110 to detect thefiducial. The augmented image information is displayed on the headmounted display 120. The fiducial is used for registration of theaugmented image information and to identify to the head mounted displaythe location in which to display the augmented image information. Theaugmented image is displayed on the head mounted display and thus isvisible only to the user. In such an embodiment, the camera sensor 110on the head mounted display detects the fiducial, registers theaugmented image with the fiducial and displays an appropriately scaledaugmented image on the head mounted display 120 to coincide with thelocation of the target.

It should be noted that as described above with reference to FIG. 2 ,the system 300 may optionally be equipped with light sources 116A, 116Band 116C.

Registration of Surgical Instruments

The augmented imaging system of the present invention can also be usedfor navigation/registration of surgical instruments. This system caninclude a distance/contour measuring device 102 as described above, aprocessor 104, a projector 108, an imaging camera 110, a look-up tablestoring information for uniquely identifying a plurality of surgicalinstruments.

In this type of system, the distance-measuring device measures thedistance and/or contour/topology of the object onto which the augmentedinformation is projected. The imaging camera captures images of thesurgical devices, compares the captured images with the informationstored in the look-up table to identify surgical instruments and theirlocation/orientation relative to the object.

The surgical tool may be provided with a fiducial to facilitateidentification and position tracking of the tool. The fiducial may beproduced with visible light spectrum or with IR fluorescent ink.Fiducials can also be constructed with other technologies, such asreflectors.

According to this embodiment, the position of the instruments is trackedwith the imaging camera 110 and augmented data in the form of guidanceinformation is projected 108 onto the object (patient), aiding the userwith properly positioning the surgical tool. The augmented informationmay include an image displayed either directly on the tool or next tothe tool by the projector providing real time feedback on how toreposition the tool, for example.

In each of the augmented imaging systems described herein, the systemassesses the distance to the object and the skew angle between theimaging camera/projector and the physical object and is therefore ableto project various messages to the user. For example, if thecamera/projector is too low or too high, a message or image can beprojected to assist the user with positioning. This can also be appliedwhen the depth sensing camera/projector is scanning an object.

The rest of the system 300 operates the same as the structured lightembodiment, and may include any combination of lights 116A, 116B, 116Cand 117.

Endoscope with Distance Measurement and Scaling

For endoscopic fluorescence imaging applications it would be desirableto determine the distance between the distal end of the endoscope andthe fluorescent target. FIG. 4 is a block diagram of an endoscopicsystem 400 that utilizes the assessed distance to optimize the intensityof fluorescent emission. A distance-measuring device 102 or thestructured light source 117 and camera sensor 110 can be utilized, forexample, to automatically adjust the intensity of the projector 108 orlights 116B/116C used for excitation.

The distance-measuring device 102 (or structured light source 117, orprojector 108) may operate through the endoscope tube and measure thedistance from the distal end of the endoscope to the surgical site.Alternatively, the distance measurement assessment could be accomplishedthrough separate optics than those of the endoscope. Distance can bemeasured, for example, by using structured light 117 and an additionalcamera/sensor 110 that is separate from the endoscope camera. Thisalternative may prove to be beneficial in that the pattern or signalused for distance sensing is not distorted by the endoscope optics.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. It is not intendedthat the invention be limited by the specific examples provided withinthe specification. While the invention has been described with referenceto the aforementioned specification, the descriptions and illustrationsof the embodiments herein are not meant to be construed in a limitingsense. Numerous variations, changes, and substitutions will now occur tothose skilled in the art without departing from the invention.Furthermore, it shall be understood that all aspects of the inventionare not limited to the specific depictions, configurations or relativeproportions set forth herein which depend upon a variety of conditionsand variables. It should be understood that various alternatives to theembodiments of the invention described herein may be employed inpracticing the invention. It is therefore contemplated that theinvention shall also cover any such alternatives, modifications,variations or equivalents. It is intended that the following claimsdefine the scope of the invention and that methods and structures withinthe scope of these claims and their equivalents be covered thereby.

What is claimed is:
 1. An imaging system projecting augmentedinformation on a physical object, the imaging system comprising: aprocessor; a memory device operably connected to the processor, thememory device storing augmented image information; a projector operablycoupled to the processor, the processor configured to project augmentedimage information onto the physical object; and a distance-measuringdevice operably connected to the processor and configured to measure thedistance to the physical object, the processor using distancemeasurement information from the distance measuring device to adjustscaling of the augmented image information, the processor providing thescale adjusted augmented image information to the projector.